{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XT2oM6FHVJCf"
   },
   "source": [
    "## Enviroment:\n",
    "\n",
    "Open AI gym [CartPole v0](https://github.com/openai/gym/wiki/CartPole-v0)\n",
    "\n",
    "### Observation\n",
    "\n",
    "Type: Box(4)\n",
    "\n",
    "| Num  | Observation          | Min      | Max     |\n",
    "| :--- | :------------------- | :------- | :------ |\n",
    "| 0    | Cart Position        | -2.4     | 2.4     |\n",
    "| 1    | Cart Velocity        | -Inf     | Inf     |\n",
    "| 2    | Pole Angle           | ~ -41.8° | ~ 41.8° |\n",
    "| 3    | Pole Velocity At Tip | -Inf     | Inf     |\n",
    "\n",
    "### Actions\n",
    "\n",
    "Type: Discrete(2)\n",
    "\n",
    "| Num  | Action                 |\n",
    "| :--- | :--------------------- |\n",
    "| 0    | Push cart to the left  |\n",
    "| 1    | Push cart to the right |\n",
    "\n",
    "Note: The amount the velocity is reduced or increased is not fixed as it depends on the angle the pole is pointing. This is because the center of gravity of the pole increases the amount of energy needed to move the cart underneath it\n",
    "\n",
    "### Reward\n",
    "\n",
    "Reward is 1 for every step taken, including the termination step\n",
    "\n",
    "### Starting State\n",
    "\n",
    "All observations are assigned a uniform random value between ±0.05\n",
    "\n",
    "### Episode Termination\n",
    "\n",
    "1. Pole Angle is more than ±12°\n",
    "2. Cart Position is more than ±2.4 (center of the cart reaches the edge of the display)\n",
    "3. Episode length is greater than 200\n",
    "\n",
    "### Solved Requirements\n",
    "\n",
    "Considered solved when the average reward is greater than or equal to 195.0 over 100 consecutive trials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xzVWtGxJVJCh"
   },
   "source": [
    "## 1. gym enviroment setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "G312rzBjVJCi",
    "outputId": "6b5a4d05-d2a3-4db8-a028-64063c670966"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.01550016,  0.02469346,  0.04678063, -0.03023327])"
      ]
     },
     "execution_count": 11,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "env = gym.make(\"CartPole-v0\")\n",
    "env.reset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "80YCNcHUZYtI"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import models\n",
    "from tensorflow.keras import losses,optimizers,metrics\n",
    "from tensorflow.keras.layers import Dense, Activation, Flatten\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "from collections import deque\n",
    "import random\n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IjkusASudj9W"
   },
   "outputs": [],
   "source": [
    "ACTION_SPACE_SIZE = env.action_space.n\n",
    "REPLAY_MEMORY_SIZE = 50_000\n",
    "MIN_REPLAY_MEMORY_SIZE = 1_000\n",
    "MINIBATCH_SIZE = 64\n",
    "UPDATE_TARGET_EVERY = 5\n",
    "DISCOUNT = 0.99\n",
    "EPISODES =1000\n",
    "\n",
    "# Exploration settings\n",
    "epsilon = 1  # not a constant, going to be decayed\n",
    "EPSILON_DECAY = 0.995\n",
    "MIN_EPSILON = 0.001\n",
    "\n",
    "ep_rewards = []\n",
    "\n",
    "AGGREGATE_STATS_EVERY = 50\n",
    "MIN_EPSILON = 0.001"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "k4-a2aU0YCbL"
   },
   "source": [
    "### Plot epsilon decay "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 286
    },
    "colab_type": "code",
    "id": "41yF0b4u46gb",
    "outputId": "ec125908-397d-466a-b4a6-f68eb509f700"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fbd4d8fd710>]"
      ]
     },
     "execution_count": 14,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xt8VOWdx/HPb2YyuZMQEgIkgXAJ\nSkRAGi+ILF2v6Cq23l27arV1bWutbV/tanfrdt19rdt1V2u71mqtbW2t1norRVu81NZLUQwKyEUg\n3AOYhFsSQq4zz/4xAwaKJoRJTubM9/16zYs5z3mS+Z0c/ebkOec8x5xziIiIvwS8LkBERBJP4S4i\n4kMKdxERH1K4i4j4kMJdRMSHFO4iIj6kcBcR8SGFu4iIDyncRUR8KOTVBxcWFrry8nKvPl5EJCkt\nXrx4h3OuqKd+noV7eXk51dXVXn28iEhSMrNNvemnYRkRER9SuIuI+JDCXUTEhxTuIiI+1GO4m9nD\nZlZvZss/Yr2Z2ffNrMbMlpnZ9MSXKSIiR6I3R+4/A+Z8zPpzgYr46wbg/qMvS0REjkaP4e6cexXY\n9TFdLgQecTFvAvlmNjJRBYqIyJFLxJh7CbCl23JtvO2vmNkNZlZtZtUNDQ19+rB3Nu/mu394Hz0e\nUETkow3oCVXn3IPOuSrnXFVRUY83WB3W8q2N3P+ndaxr2Jvg6kRE/CMR4b4VKOu2XBpv6xdnVRYD\nsGBFXX99hIhI0ktEuM8Dro5fNXMK0Oic256A73tYI/MymVqaxwsrPuivjxARSXq9uRTyMWAhcIyZ\n1ZrZ9WZ2o5ndGO/yPLAeqAF+DHyx36qNO/u4ESytbWR7Y2t/f5SISFLqceIw59yVPax3wJcSVlEv\nnHPcCO5asJoXV9Zx9YzygfxoEZGkkJR3qE4YnsO4omwWaGhGROSwkjLcIXb0/ub6XezZ1+F1KSIi\ng07ShvvZlcVEoo4/vl/vdSkiIoNO0ob71NJ8ioeka2hGROQwkjbcAwHjrMpi/rymgdaOiNfliIgM\nKkkb7hAbd2/rjPLa2r5NZSAi4ldJHe4njx1GbkZId6uKiBwiqcM9HApw1qRiXlz5AR1dUa/LEREZ\nNJI63AH+bspImtq6eKNmh9eliIgMGkkf7qdVFJKbEWL+sn6bzkZEJOkkfbinh4KcXTmCFzQ0IyJy\nQNKHO8DfTRlBc1sXr9foqhkREfBJuJ82oUhDMyIi3fgi3MOhAOccN4IXV9TR3qUbmkREfBHuELtq\nprm9i9fX6qoZERHfhPvM8YXkZabxnIZmRET8E+7hUICzK4t5caWGZkREfBPu8OHQzGtrNDQjIqnN\nV+E+c0Ih+Vlp/G7ZNq9LERHxlK/CPS0Y4NzJI3lhRR0t7V1elyMi4hlfhTvAp6aNorUzwosrNVOk\niKQu34X7ieUFlORn8uySrV6XIiLiGd+FeyBgXDhtFK+t3cGOve1elyMi4gnfhTvAp04oIRJ1zF+q\nE6sikpp8Ge4Ti3OpHDmEZ5Yo3EUkNfky3AE+dcIolm7Zw4YdLV6XIiIy4Hwb7nOnlmAGz76rE6si\nknp8G+4j8jI4dfwwnl2yFeec1+WIiAwo34Y7wIXTSti0cx9LtuzxuhQRkQHl63CfM3kE6aEAz2ho\nRkRSjK/DfUhGGmcfN4LfLtlGW6dmihSR1NGrcDezOWa22sxqzOzWw6wfbWavmNm7ZrbMzM5LfKl9\nc1lVKY2tnby0StMRiEjq6DHczSwI3AecC1QCV5pZ5SHd/gV4wjl3AnAF8MNEF9pXp44vpCQ/kyeq\na70uRURkwPTmyP0koMY5t9451wE8Dlx4SB8HDIm/zwMGzd1DwYBx8fQSXlvbwLY9rV6XIyIyIHoT\n7iXAlm7LtfG27r4DfMbMaoHngS8f7huZ2Q1mVm1m1Q0NDX0ot28u+UQZzsHT7+joXURSQ6JOqF4J\n/Mw5VwqcB/zCzP7qezvnHnTOVTnnqoqKihL00T0bPSyLGeOG8UR1LdGornkXEf/rTbhvBcq6LZfG\n27q7HngCwDm3EMgAChNRYKJcWlXK5l37WLRxl9eliIj0u96E+9tAhZmNNbMwsROm8w7psxk4A8DM\nJhEL94Ebd+mFcyePJCc9xG90YlVEUkCP4e6c6wJuAhYAq4hdFbPCzO4ws7nxbl8HPm9mS4HHgGvd\nILvnPzMc5IKpo3j+ve00t3V6XY6ISL8K9aaTc+55YidKu7fd3u39SmBmYktLvEurSnls0WaeW7ad\nK04a7XU5IiL9xtd3qB7qhLJ8Kobn8NiizV6XIiLSr1Iq3M2Mq04ezdLaRt6rbfS6HBGRfpNS4Q7w\n6emlZKYF+dWiTV6XIiLSb1Iu3PMy05g7dRTPvruNJp1YFRGfSrlwB7jqlNG0dkb0lCYR8a2UDPcp\npfkcX5LHo29u1lOaRMSXUjLcAT5zymhW1zVTvWm316WIiCRcyob7BVNHkZse4tE3dWJVRPwnZcM9\nKxziouklPP/eB+xq6fC6HBGRhErZcAe46pQxdESiPFG9pefOIiJJJKXDfWJxLiePLeAXCzfRFYl6\nXY6ISMKkdLgDXHfaWLbuaeXFlXrGqoj4R8qH+5mTiikryOSnb2z0uhQRkYRJ+XAPBoxrZpSzaOMu\nlm/VfDMi4g8pH+4Al51YRnY4yMNvbPC6FBGRhFC4A0My0rjkE6XMX7qdhuZ2r8sRETlqCve4a04t\npyMS5dG3dFOTiCQ/hXvcuKIcTj92OL98cxPtXRGvyxEROSoK924+O7OcHXs7+N3S7V6XIiJyVBTu\n3Zw2oZBjinP58avrNVukiCQ1hXs3ZsY/zh7H6rpmXlld73U5IiJ9pnA/xAVTRzEqL4Mf/Xm916WI\niPSZwv0QacEA188ax6INu3hns+Z6F5HkpHA/jCtOLCMvM40f/Wmd16WIiPSJwv0wstNDXDNjDC+u\nqqOmfq/X5YiIHDGF+0e4+tRywsEAD76qo3cRST4K949QmJPOZVVlPPPuVj5obPO6HBGRI6Jw/xif\nnzWOSNTx49d05YyIJBeF+8cYPSyLT00r4dG3NmlCMRFJKgr3Htx0+gQ6uqI8pKN3EUkiCvcejCvK\nYe7UUTyycBM79+roXUSSQ6/C3czmmNlqM6sxs1s/os9lZrbSzFaY2a8SW6a3bjq9grauCD9+TQ/z\nEJHk0GO4m1kQuA84F6gErjSzykP6VAC3ATOdc8cBt/RDrZ6ZMDyHC6aM4pGFG9nV0uF1OSIiPerN\nkftJQI1zbr1zrgN4HLjwkD6fB+5zzu0GcM75btatL58+gdbOCD95XWPvIjL49SbcS4At3ZZr423d\nTQQmmtkbZvammc053DcysxvMrNrMqhsaGvpWsUcqinM57/iR/Pwvm9izT0fvIjK4JeqEagioAD4J\nXAn82MzyD+3knHvQOVflnKsqKipK0EcPnJtPr6Clo4sHX9XRu4gMbr0J961AWbfl0nhbd7XAPOdc\np3NuA7CGWNj7yjEjcpk7dRQ/fWMj9U26a1VEBq/ehPvbQIWZjTWzMHAFMO+QPs8SO2rHzAqJDdP4\n8vD2a2dNpDMS5Qd/rPG6FBGRj9RjuDvnuoCbgAXAKuAJ59wKM7vDzObGuy0AdprZSuAV4BvOuZ39\nVbSXxgzL5vITy3hs0WY279zndTkiIodlXj0rtKqqylVXV3vy2UerrqmN2Xe9wrmTR3LP5dO8LkdE\nUoiZLXbOVfXUT3eo9kHxkAyuPXUszy7ZyvsfNHldjojIX1G499EXZo8nJz3E/yxY7XUpIiJ/ReHe\nR3lZadw4ezwvraqneuMur8sRETmIwv0ofHZmOcNz0/mP51YRjXpz7kJE5HAU7kchKxziG+ccw5It\ne/jdsm1elyMicoDC/ShdPL2U40YN4b//sJq2zojX5YiIAAr3oxYIGP/yd5Vs3dPKT17XlMAiMjgo\n3BNgxvhhnF1ZzA9fqaG+WdMSiIj3FO4J8q3zJtERiXL3C2u8LkVEROGeKOWF2Vwzo5xfV29h5Tbd\n2CQi3lK4J9CXT69gaFaY23+7HK+mdRARAYV7QuVlpXHrucdSvWk3T79z6KzIIiIDR+GeYJdML2X6\n6Hzu/P0qGls7vS5HRFKUwj3BAgHjjgsns6ulg3te1MlVEfGGwr0fTC7J4x9OGcMjCzeyYluj1+WI\nSApSuPeTr519DEOzwnz72eWad0ZEBpzCvZ/kZaZx23mTeGfzHn5dvcXrckQkxSjc+9HF00s4ZVwB\n//n8Kur0QG0RGUAK935kZtx50RQ6uqLc/tvlXpcjIilE4d7PxhZmc8uZE1mwoo4/LN/udTkikiIU\n7gPgc7PGUjlyCN/+7Qoa9+nadxHpfwr3AZAWDPDfl0xhV0sHd/5+ldfliEgKULgPkMkleXxu1lge\nf3sLr6/d4XU5IuJzCvcB9NUzJzKuKJtvPLlUUxOISL9SuA+gjLQgd182jfrmdv7tdyu8LkdEfEzh\nPsCmleXzpb+dwNPvbNXVMyLSbxTuHvjy6ROYXDKEbz2znIbmdq/LEREfUrh7IC0Y4J7LprG3vYtv\nPfOeHuwhIgmncPdIRXEu3zznGF5cWcdjizT3jIgklsLdQ9fNHMusikL+7XcrWFPX7HU5IuIjCncP\nBQLG3ZdNIzcjjZt+9Q6tHRGvSxIRn+hVuJvZHDNbbWY1Znbrx/S72MycmVUlrkR/K8pN5+7LprKm\nbi93zF/pdTki4hM9hruZBYH7gHOBSuBKM6s8TL9c4CvAW4ku0u/+ZmIRN84ez2OLNjN/2TavyxER\nH+jNkftJQI1zbr1zrgN4HLjwMP3+HfguoInL++DrZ09kWlk+tz31Hpt37vO6HBFJcr0J9xKg++Uc\ntfG2A8xsOlDmnHvu476Rmd1gZtVmVt3Q0HDExfpZWjDAD648AQy++KvFtHVq/F1E+u6oT6iaWQC4\nG/h6T32dcw8656qcc1VFRUVH+9G+U1aQxfcun8byrU388zPLdf27iPRZb8J9K1DWbbk03rZfLjAZ\n+JOZbQROAebppGrfnDGpmJvPqOCpd2r55VubvS5HRJJUb8L9baDCzMaaWRi4Api3f6VzrtE5V+ic\nK3fOlQNvAnOdc9X9UnEKuOWMCv72mCLu+N0KFm/a7XU5IpKEegx351wXcBOwAFgFPOGcW2Fmd5jZ\n3P4uMBUFAsb3Lj+BkXmZfPHRxdQ36xy1iBwZ82pct6qqylVX6+D+46zc1sRF97/B8SV5/PJzJ5Me\nCnpdkoh4zMwWO+d6HPbWHaqDWOWoIdx1yVTe3rib257WBGMi0nshrwuQj3fB1FGsb2jhnpfWML4o\nhy/97QSvSxKRJKBwTwI3nzGB9Tv2cteC1YwtzOa840d6XZKIDHIalkkCZsZ3L57C9NH5fO2JJSzd\nssfrkkRkkFO4J4mMtCAPXl1FYU46n3ukmi27NEWBiHw0hXsSKcxJ56fXnkh7Z4SrH17Ezr16RJ+I\nHJ7CPclUFOfy8LUnsm1PK9f97G1a2ru8LklEBiGFexKqKi/gvr+fzvJtTdz4y8V0dEW9LklEBhmF\ne5I6s7KYOz99PK+t3cE3nlxKNKpr4EXkQ7oUMolddmIZDXvbuWvBanIzQvz7hZMxM6/LEpFBQOGe\n5L74yfE0tXXywJ/XEw4G+fb5kxTwIqJwT3Zmxq1zjqWjK8rDb2wgHArwT3OOUcCLpDiFuw+YGbef\nX0lnJMqP/ryO9FCAr5410euyRMRDCnefMDPumDuZjq4o9768lmDA+PLpE3QEL5KiFO4+EggYd140\nhUgU7n5xDa2dEb55joZoRFKRwt1nggHjrkumkJEW4P4/raO1I8Lt51cSCCjgRVKJwt2HAgHjPz41\nmaxwkB+/toHWjgj/edHxBBXwIilD4e5TZsa3zptEZjjE919ey77OCP976VTCId23JpIKFO4+ZmZ8\n7ayJZIWD/Nfv32fPvg5+eNV0cjPSvC5NRPqZDuNSwI2zx3PXJVP4y7qdXP7Am9Q16YHbIn6ncE8R\nl1aV8ZNrqti4s4WLfvgXauqbvS5JRPqRwj2FfPKY4fz6hhm0d0W5+P6FLNqwy+uSRKSfKNxTzPGl\neTzzxVMZlh3mqofe5Im3t3hdkoj0A4V7CioryOKZL87k5LHD+OZTy/j3+SvpimhOeBE/UbinqLys\nNH722RO59tRyfvL6Bq77eTWNrZ1elyUiCaJwT2GhYIDvzD2OOy86noXrdvDp+95gbZ1OtIr4gcJd\nuPKk0Tz6uVNoautk7v+9wbPvbvW6JBE5Sgp3AeCksQU8d/Msji/J45ZfL+G2p9+jrTPidVki0kcK\ndzmgeEgGv/r8ydw4ezyPLdrMxff/hU07W7wuS0T6QOEuBwkFA9x67rE8dHUVW3bt4/wfvM5vl2iY\nRiTZKNzlsM6sLOa5m2dRMTyHrzy+hK88/q6uphFJIr0KdzObY2arzazGzG49zPqvmdlKM1tmZi+b\n2ZjElyoDrawgiyf+cQZfO2si85dt59zvvcrCdTu9LktEeqHHcDezIHAfcC5QCVxpZpWHdHsXqHLO\nTQGeBP470YWKN0LBADefUcFTXziV9LQgf//Qm9z5/CqdbBUZ5Hpz5H4SUOOcW++c6wAeBy7s3sE5\n94pzbl988U2gNLFlitemleXz3M2nceVJo3ng1fWcd+9rmptGZBDrTbiXAN0nIKmNt32U64HfH26F\nmd1gZtVmVt3Q0ND7KmVQyAqH+M9PH88vrz+ZjkiUyx5YyLefXU5zm8biRQabhJ5QNbPPAFXAXYdb\n75x70DlX5ZyrKioqSuRHywA6raKQF776N1w3cyy/fGsT59zzKn98v87rskSkm96E+1agrNtyabzt\nIGZ2JvDPwFznXHtiypPBKisc4vYLKnnqC6eSnR7iup9V8/lHqtmya1/PXywi/a434f42UGFmY80s\nDFwBzOvewcxOAB4gFuz1iS9TBqvpo4fy3M2z+Kc5x/JGzQ7OvPvP3PvSWp1wFfFYj+HunOsCbgIW\nAKuAJ5xzK8zsDjObG+92F5AD/MbMlpjZvI/4duJD4VCAL3xyPC9/fTZnVhZzz0trOPueV3lpZR3O\nOa/LE0lJ5tX/fFVVVa66utqTz5b+9UbNDv513gpq6vdy6vhhfOu8SUwuyfO6LBFfMLPFzrmqnvrp\nDlVJuJkTCnn+5ll854JKVm1v4vwfvM4tj7+r8XiRAaQjd+lXTW2d/OhP6/jJ6xtwDq6dWc6Ns8dT\nkB32ujSRpNTbI3eFuwyI7Y2t/O8La3jqnVqy0oJcc2o5n581jqEKeZEjonCXQWltXTP3vryW597b\nTlZakGtnxkI+P0shL9IbCncZ1NbEQ/7597aTHQ5x9YwxXDuznOG5GV6XJjKoKdwlKaz+oJnv/zEW\n8mmBABdNL+Fzs8YxYXiO16WJDEoKd0kqG3e08NDr6/lNdS3tXVHOnFTMjbPHUVVe4HVpIoOKwl2S\n0o697TyycBO/WLiR3fs6mVKaxz+cMoYLpo4iIy3odXkinlO4S1Lb19HFU4tr+fnCTdTU7yU/K43L\nqsq46uTRjBmW7XV5Ip5RuIsvOOd4c/0ufvHmRhasqCPqHLMnFnF5VRlnTComHNJ9eJJaFO7iOx80\ntvHYos08/vZm6praGZqVxoXTSri0qpTjRml6A0kNCnfxra5IlNdqdvDk4lpeXFFHRyTKpJFDuPQT\npVwwdRRFuelelyjSbxTukhL27Otg3tJtPLm4lmW1jQQMThk3jAumjmLOcSN0B6z4jsJdUs6aumbm\nL93G/GXbWb+jhVDAmDmhkPOnjOTs40aQl5nmdYkiR03hLinLOceKbU3MX7ad+cu2Ubu7lVDAOHlc\nAWdOKubMScWUFWR5XaZInyjcRYgF/dLaRhas+ICXVtaxtn4vAMeOyOWsyljQH1+SRyBgHlcq0jsK\nd5HD2LCjhZdX1fHCyjqqN+4i6mBYdpjTKgqZVVHErIpCiodofhsZvBTuIj3Y3dLBK6vreXVNA6/X\n7GDH3g4AjinOZVZFIbMmFnFi+VCywiGPKxX5kMJd5AhEo45VHzTx2todvL52B4s27qKjK0ooYBxf\nmsdJYws4eWwBnxhToBOz4imFu8hRaO2IsGjjLt5av5NFG3axtHYPnRGHGUwaMYSTxhZw0tgCThid\nz4ghGZhpzF4GhsJdJIHaOiO8u3kPizbsYtHGnSzetJu2zigAw3PTmVaWz7TR+Uwrzef40jxyM3R0\nL/2jt+GuwUSRXshICzJj/DBmjB8GVNDRFWXFtkaWbtnDki17WFrbyAsr6wAwg4rhOUwpzee4UUOY\nNDL20nCODCSFu0gfhEMBThg9lBNGDz3QtmdfB0trG1myeQ9La/fwp9X1PLm49sD60qGZVMaDvnLU\nECpHDqF0aKaGdKRfKNxFEiQ/K8zsiUXMnlh0oK2+uY2V25pYub2JVdubWbmtkRdX1bF/NDQ7HGT8\n8BwmFOUwfngOFcNzmDA8h9EFWYSCmvFS+k7hLtKPhudmMPyYDD55zPADba0dEVbXNbNyWxNr6ppZ\n17CXhet38vS7Ww/0CQcDlBdmMWF4DuMKcxg9LIvyYdmMGZbF8Nx0He1LjxTuIgMsMxyMnYAtyz+o\nvbmtk3UNLdTU72VtfTPr6veyclsTC1bUEYl+eOFDZlqQ0QVZjB6WxZiCLMYUZjOmIIvSoZmMys/U\nE6sEULiLDBq5GWmHDf3OSJStu1vZtGsfm3a2sGnnPjbt3MfGHS28uqaB9q7oQf2HZYcZlZ/JyLwM\nRuVnUpIfC/1R+RmU5GdSmJOu6RZSgMJdZJBLCwYoL8ymvDAbKDpoXTTqqG9uZ9POFrbuaWXbnla2\nNbaxbU8rG3e28EbNDlo6Igd9TShgFOWmMzw3naLcjAPvhw9Jjw0j5aZTFH+ladw/aSncRZJYIGCM\nyMtgRN7h58NxztHU1hUL/W7hX9/UTn1zG7W79/Hu5t3sbOk47NcXZIcpzAkzNCvMsP3/ZocZmh2m\nIP7qvk5DQoOHwl3Ex8yMvMw08jLTmDRyyEf264xE2bG3PR767TQ0x8K/vrmdnXvb2d3SyeoPmtm9\nr5Pd+zr4qHsfs8NBhmaHyc9KY0hG/JUZIi9z//tYLUMyQwcvZ6SRkRbQieIEUriLCGnBACPzMhmZ\nl9lj30jU0djaya6WjgOv3fs6DlpubO2kqbWT9Tv20tTaRWNrJ62dkY/9vmlBIzcjjez0INnhENnp\nsVdOepCscIic9BDZB70PkR0OdusXIiscJCMtSGY4SEYokNKXk/Yq3M1sDnAvEAQecs791yHr04FH\ngE8AO4HLnXMbE1uqiAwGwYAdGJI5Eh1dUZrbOmlq6zoQ/k1tnQfCv6mtk+a2Tva1R9jb3kVLR6x9\n+55WWtq74m2Rg64c6kla0MgIBckIB8lIC5CZFgv//a/MtED83+5tsb7poQBpoQDhYIBwKLYcDgUI\nB4OEQwHSgvZhe7yte3s46O1fIj2Gu5kFgfuAs4Ba4G0zm+ecW9mt2/XAbufcBDO7AvgucHl/FCwi\nySkcCjAsJ51hOX1/gLlzjvauKC3tXezriP8SiId+S/x9W1eUto4IbZ0RWuOvts4obZ0ftrV1Rmhs\n7aSuMUJbV4TWeP+2zigdkWjPhfR2m+O/GA4N/VvOnMgFU0cl7HMOpzdH7icBNc659QBm9jhwIdA9\n3C8EvhN//yTwf2ZmzqtZyUTEl8zswBH2sH76jEjU0dYZob0rSsf+V+SQf7uidEQidHS5g9u6InRE\nonRG3CFfHznwvjPqyM/q/3mGehPuJcCWbsu1wMkf1cc512VmjcAwYEf3TmZ2A3ADwOjRo/tYsohI\n/wkGLD6O73UlR2dAzzY45x50zlU556qKiop6/gIREemT3oT7VqCs23JpvO2wfcwsBOQRO7EqIiIe\n6E24vw1UmNlYMwsDVwDzDukzD7gm/v4S4I8abxcR8U6PY+7xMfSbgAXELoV82Dm3wszuAKqdc/OA\nnwC/MLMaYBexXwAiIuKRXl3n7px7Hnj+kLbbu71vAy5NbGkiItJXqXv7loiIjyncRUR8SOEuIuJD\n5tVFLWbWAGzq45cXcsgNUilA25watM2p4Wi2eYxzrscbhTwL96NhZtXOuSqv6xhI2ubUoG1ODQOx\nzRqWERHxIYW7iIgPJWu4P+h1AR7QNqcGbXNq6PdtTsoxdxER+XjJeuQuIiIfI+nC3czmmNlqM6sx\ns1u9ridRzKzMzF4xs5VmtsLMvhJvLzCzF81sbfzfofF2M7Pvx38Oy8xsurdb0DdmFjSzd81sfnx5\nrJm9Fd+uX8cnq8PM0uPLNfH15V7W3Vdmlm9mT5rZ+2a2ysxmpMA+/mr8v+nlZvaYmWX4cT+b2cNm\nVm9my7u1HfG+NbNr4v3Xmtk1h/us3kiqcO/2yL9zgUrgSjOr9LaqhOkCvu6cqwROAb4U37ZbgZed\ncxXAy/FliP0MKuKvG4D7B77khPgKsKrb8neBe5xzE4DdxB7hCN0e5QjcE++XjO4F/uCcOxaYSmzb\nfbuPzawEuBmocs5NJjb54P5HcfptP/8MmHNI2xHtWzMrAP6V2AORTgL+df8vhCPmnEuaFzADWNBt\n+TbgNq/r6qdt/S2x59auBkbG20YCq+PvHwCu7Nb/QL9keRF7NsDLwOnAfMCI3dgROnR/E5uVdEb8\nfSjez7zehiPc3jxgw6F1+3wf739KW0F8v80HzvHrfgbKgeV93bfAlcAD3doP6nckr6Q6cufwj/wr\n8aiWfhP/U/QE4C2g2Dm3Pb7qA6A4/t4PP4vvAd8E9j+ReBiwxznXFV/uvk0HPcoR2P8ox2QyFmgA\nfhofinrIzLLx8T52zm0F/gfYDGwntt8W4+/93N2R7tuE7fNkC3ffM7Mc4CngFudcU/d1Lvar3BeX\nN5nZ+UC9c26x17UMoBAwHbjfOXcC0MKHf6YD/trHAPEhhQuJ/WIbBWTz10MXKWGg922yhXtvHvmX\ntMwsjViwP+qcezreXGdmI+PrRwL18fZk/1nMBOaa2UbgcWJDM/cC+fFHNcLB2+SHRznWArXOubfi\ny08SC3u/7mOAM4ENzrkG51wn8DSxfe/n/dzdke7bhO3zZAv33jzyLymZmRF7otUq59zd3VZ1f4Th\nNcTG4ve3Xx0/634K0Njtz7/axVoVAAABEElEQVRBzzl3m3Ou1DlXTmw//tE5dxXwCrFHNcJfb29S\nP8rROfcBsMXMjok3nQGsxKf7OG4zcIqZZcX/G9+/zb7dz4c40n27ADjbzIbG/+o5O9525Lw+AdGH\nExbnAWuAdcA/e11PArfrNGJ/si0DlsRf5xEbb3wZWAu8BBTE+xuxK4fWAe8RuxrB8+3o47Z/Epgf\nfz8OWATUAL8B0uPtGfHlmvj6cV7X3cdtnQZUx/fzs8BQv+9j4N+A94HlwC+AdD/uZ+AxYucVOon9\nlXZ9X/YtcF18+2uAz/a1Ht2hKiLiQ8k2LCMiIr2gcBcR8SGFu4iIDyncRUR8SOEuIuJDCncRER9S\nuIuI+JDCXUTEh/4f95m5BOAGZZIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "recorder = {\"epsode\":[],\"epsilon\":[]}\n",
    "\n",
    "for epsode in range(EPISODES):\n",
    "    if epsilon > MIN_EPSILON:\n",
    "        epsilon *= EPSILON_DECAY\n",
    "        epsilon = max(MIN_EPSILON, epsilon)\n",
    "    \n",
    "    recorder[\"epsode\"].append(epsode)\n",
    "    recorder[\"epsilon\"].append(epsilon)\n",
    "\n",
    "plt.plot(recorder[\"epsode\"], recorder[\"epsilon\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2bEwIRskgFVq"
   },
   "source": [
    "## 2.  create_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jjosMYBJYk4J"
   },
   "outputs": [],
   "source": [
    "def create_model():\n",
    "  model = models.Sequential()\n",
    "  \n",
    "#   model.add(Flatten(input_shape=(2,)))\n",
    "  model.add(Dense(16 ,input_shape=(env.observation_space.shape)))\n",
    "  model.add(Activation('relu'))\n",
    "  model.add(Dense(16))\n",
    "  model.add(Activation('relu'))\n",
    "  model.add(Dense(16))\n",
    "  model.add(Activation('relu'))\n",
    "  model.add(Dense(ACTION_SPACE_SIZE))\n",
    "  model.add(Activation('linear'))\n",
    "  \n",
    "  print(model.summary())\n",
    "  \n",
    "  model.compile(loss = 'mse', optimizer=Adam(lr=0.001),metrics=['accuracy'])\n",
    "  \n",
    "  return model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lTRr9PZPZ1-1"
   },
   "source": [
    "## 3. Agent Class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "w_7sLLVzZ1Tj"
   },
   "outputs": [],
   "source": [
    "class DQNAgent:\n",
    "\n",
    "  def __init__(self):\n",
    "    \n",
    "    # Replay memory\n",
    "    self.replay_memory = deque(maxlen=REPLAY_MEMORY_SIZE)\n",
    "    \n",
    "    # Prediction Network (the main Model)\n",
    "    self.model_prediction = create_model()\n",
    "    \n",
    "    # Target Network\n",
    "    self.model_target = create_model()\n",
    "    self.model_target.set_weights(self.model_prediction.get_weights())\n",
    "    \n",
    "   \n",
    "    \n",
    "    # Used to count when to update target network with prediction network's weights\n",
    "    self.target_update_counter = 0\n",
    "    \n",
    "    \n",
    "  \n",
    "  # Adds step's data to a memory replay array\n",
    "  # (current_state, action, reward, next_state, done)\n",
    "  def update_replay_memory(self, transition):\n",
    "    self.replay_memory.append(transition)\n",
    "  \n",
    "  \n",
    "  \n",
    "  # Queries prediction network for Q values given current observation space (environment state)\n",
    "  def get_qs(self, state):\n",
    "    return self.model_prediction.predict(np.array(state).reshape(-1, *state.shape))[0]\n",
    "  \n",
    "  def train(self, terminal_state, step):\n",
    "    \n",
    "    if len(self.replay_memory) < MIN_REPLAY_MEMORY_SIZE:\n",
    "      return\n",
    "    \n",
    "    minibatch = random.sample(self.replay_memory, MINIBATCH_SIZE)\n",
    "    \n",
    "    # Get current states from minibatch, then query NN model_prediction for current Q values\n",
    "    current_states = np.array([transition[0] for transition in minibatch])\n",
    "    current_qs_list = self.model_prediction.predict(current_states)   \n",
    "    \n",
    "    # Get next_states from minibatch, then query NN model_target for target Q values\n",
    "    # When using target network, query it, otherwise main network should be queried\n",
    "    next_states = np.array([transition[3] for transition in minibatch])\n",
    "    target_qs_list = self.model_target.predict(next_states)\n",
    "    \n",
    "    X = []\n",
    "    y = []\n",
    "    \n",
    "    # Now we need to enumerate our batches\n",
    "    for index, (current_state, action, reward, new_current_state, done) in enumerate(minibatch):\n",
    "\n",
    "        # If not a terminal state, get new q from future states, otherwise set it to 0\n",
    "        # almost like with Q Learning, but we use just part of equation here\n",
    "        if not done:\n",
    "            max_target_q = np.max(target_qs_list[index])\n",
    "            new_q = reward + DISCOUNT * max_target_q\n",
    "        else:\n",
    "            new_q = reward\n",
    "\n",
    "        # Update Q value for given state\n",
    "        current_qs = current_qs_list[index]\n",
    "        current_qs[action] = new_q\n",
    "\n",
    "        # And append to our training data\n",
    "        X.append(current_state)\n",
    "        y.append(current_qs)    \n",
    "    \n",
    "    # Fit on all samples as one batch, log only on terminal state\n",
    "    self.model_prediction.fit(np.array(X), np.array(y), batch_size=MINIBATCH_SIZE, verbose=0, shuffle=False if terminal_state else None)\n",
    "    \n",
    "    # Update target network counter every episode\n",
    "    if terminal_state:\n",
    "        self.target_update_counter +=1\n",
    "    \n",
    "    # If counter reaches set value, update target network with weights of main network\n",
    "    if self.target_update_counter > UPDATE_TARGET_EVERY:\n",
    "      self.model_target.set_weights(self.model_prediction.get_weights())\n",
    "      self.target_update_counter = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 874
    },
    "colab_type": "code",
    "id": "TUDDhaxOadLa",
    "outputId": "80dc2cb0-617c-4d34-ca6a-e18a31a4465f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_8 (Dense)              (None, 16)                80        \n",
      "_________________________________________________________________\n",
      "activation_8 (Activation)    (None, 16)                0         \n",
      "_________________________________________________________________\n",
      "dense_9 (Dense)              (None, 16)                272       \n",
      "_________________________________________________________________\n",
      "activation_9 (Activation)    (None, 16)                0         \n",
      "_________________________________________________________________\n",
      "dense_10 (Dense)             (None, 16)                272       \n",
      "_________________________________________________________________\n",
      "activation_10 (Activation)   (None, 16)                0         \n",
      "_________________________________________________________________\n",
      "dense_11 (Dense)             (None, 2)                 34        \n",
      "_________________________________________________________________\n",
      "activation_11 (Activation)   (None, 2)                 0         \n",
      "=================================================================\n",
      "Total params: 658\n",
      "Trainable params: 658\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Model: \"sequential_3\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_12 (Dense)             (None, 16)                80        \n",
      "_________________________________________________________________\n",
      "activation_12 (Activation)   (None, 16)                0         \n",
      "_________________________________________________________________\n",
      "dense_13 (Dense)             (None, 16)                272       \n",
      "_________________________________________________________________\n",
      "activation_13 (Activation)   (None, 16)                0         \n",
      "_________________________________________________________________\n",
      "dense_14 (Dense)             (None, 16)                272       \n",
      "_________________________________________________________________\n",
      "activation_14 (Activation)   (None, 16)                0         \n",
      "_________________________________________________________________\n",
      "dense_15 (Dense)             (None, 2)                 34        \n",
      "_________________________________________________________________\n",
      "activation_15 (Activation)   (None, 2)                 0         \n",
      "=================================================================\n",
      "Total params: 658\n",
      "Trainable params: 658\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "agent = DQNAgent()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Bpa2BCspawS3"
   },
   "source": [
    "## 4. Train the agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "I7j_aM56tO2_"
   },
   "outputs": [],
   "source": [
    "aggr_ep_rewards = {'ep':[],'avg':[],'min':[],'max':[]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "Xmoxp-PEjN3s",
    "outputId": "323e47ce-b7b4-4e4b-8c10-b3ede1fb80d0"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|##########| 1000/1000 [02:08<00:00,  6.80episodes/s]\n"
     ]
    }
   ],
   "source": [
    "# Iterate over episodes\n",
    "for episode in tqdm(range(1, EPISODES + 1), ascii=True, unit='episodes'):\n",
    "\n",
    "\n",
    "    # Restarting episode - reset episode reward and step number\n",
    "    episode_reward = 0\n",
    "    step = 1\n",
    "\n",
    "    # Reset environment and get initial state\n",
    "    current_state = env.reset()\n",
    "\n",
    "    # Reset flag and start iterating until episode ends\n",
    "    done = False\n",
    "    while not done:\n",
    "\n",
    "        # This part stays mostly the same, the change is to query a model for Q values\n",
    "        if np.random.random() > epsilon:\n",
    "            # Get action from Q table\n",
    "            action = np.argmax(agent.get_qs(current_state))\n",
    "        else:\n",
    "            # Get random action\n",
    "            action = np.random.randint(0, ACTION_SPACE_SIZE)\n",
    "\n",
    "        next_state, reward, done, _ = env.step(action)\n",
    "\n",
    "        # Transform new continous state to new discrete state and count reward\n",
    "        episode_reward += reward\n",
    "\n",
    "#         if SHOW_PREVIEW and not episode % AGGREGATE_STATS_EVERY:\n",
    "#             env.render()\n",
    "\n",
    "        # Every step we update replay memory and train main network\n",
    "        agent.update_replay_memory((current_state, action, reward, next_state, done))\n",
    "        agent.train(done, step)\n",
    "\n",
    "        current_state = next_state\n",
    "        step += 1\n",
    "\n",
    "    # Append episode reward to a list and log rewards (every given number of episodes)\n",
    "    ep_rewards.append(episode_reward)\n",
    "    if not episode % AGGREGATE_STATS_EVERY or episode == 1:\n",
    "        average_reward = sum(ep_rewards[-AGGREGATE_STATS_EVERY:])/len(ep_rewards[-AGGREGATE_STATS_EVERY:])\n",
    "        min_reward = min(ep_rewards[-AGGREGATE_STATS_EVERY:])\n",
    "        max_reward = max(ep_rewards[-AGGREGATE_STATS_EVERY:])\n",
    "        \n",
    "        aggr_ep_rewards['ep'].append(episode)\n",
    "        aggr_ep_rewards['avg'].append(average_reward)\n",
    "        aggr_ep_rewards['min'].append(min_reward)\n",
    "        aggr_ep_rewards['max'].append(max_reward)\n",
    "        \n",
    "#         agent.tensorboard.update_stats(reward_avg=average_reward, reward_min=min_reward, reward_max=max_reward, epsilon=epsilon)\n",
    "\n",
    "        # Save model, but only when min reward is greater or equal a set value\n",
    "#         if min_reward >= MIN_REWARD:\n",
    "#             agent.model.save(f'models/{MODEL_NAME}__{max_reward:_>7.2f}max_{average_reward:_>7.2f}avg_{min_reward:_>7.2f}min__{int(time.time())}.model')\n",
    "\n",
    "    # Decay epsilon\n",
    "    if epsilon > MIN_EPSILON:\n",
    "        epsilon *= EPSILON_DECAY\n",
    "        epsilon = max(MIN_EPSILON, epsilon)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QQ22Ry_q4mQS"
   },
   "source": [
    "## 5. Plot episodes vs rewards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 300
    },
    "colab_type": "code",
    "id": "G1TtpQaZVJDA",
    "outputId": "4597ec7d-1551-43a9-d752-bd5df421914b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Rewards')"
      ]
     },
     "execution_count": 25,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3Xd4m9XZ+PHv7T1iO15yduzsOMNW\nBhACNMwCPwijzBZKgJbSl5ZRKIWOl/Z9S1+6oJsyStmjzAbasncIgSRSEsdZTuIkTmzLI95T1vn9\nIck4jodsa9m+P9fly9ajRzpHfiTdz3PGfcQYg1JKKdWXiFBXQCmlVPjTYKGUUqpfGiyUUkr1S4OF\nUkqpfmmwUEop1S8NFkoppfqlwUIppVS/NFgopZTqlwYLpZRS/YoKdQX8JSMjw2RnZ4e6GkopNaxs\n2LCh0hiT2d9+IyZYZGdns379+lBXQymlhhUR2efLftoMpZRSql8aLJRSSvVLg4VSSql+jZg+i560\nt7dTUlJCS0tLqKsSMHFxcUyaNIno6OhQV0UpNYKN6GBRUlJCUlIS2dnZiEioq+N3xhiqqqooKSkh\nJycn1NVRSo1gAWuGEpFHRMQhIgVdtl0sIltFxCUiS/p47JkiskNEikTkjsHWoaWlhfT09BEZKABE\nhPT09BF95aSUCg+B7LN4FDiz27YC4ELgw94eJCKRwJ+Bs4Bc4HIRyR1sJUZqoPAa6a9PKRUeAtYM\nZYz5UESyu23bBv1+wR0DFBlj9nj2fRY4DygMSEVDqK61jpaOwV8VJEQlMCZmjB9r1L92VztPFj5J\nY3tjUMsdziIjIvnKzK9gSbCEuio+a2xv5M3iNzl/xvlBPyFZc3AN4xPHM23stKCWW9lcyfM7n6fD\n1RHUcsH9nXjOtHOYmjw16GX7Khz7LCYCB7rcLgGO7WlHEbkOuA5gypQpga+ZH7mMi5KGEoayBnpU\nRBSzUmf5sVb9W3NwDfduuBcAQa9qfGEwtDpbuXnxzaGuis9eKXqFez67h5yUHPIt+UEr1+lycsv7\nt3Ds+GP54yl/DFq5AP/Y8Q/u33R/SN7XBsOhhkPcfcLdQS/bV+EYLHxmjHkQeBBgyZIlg//WDYEW\nZwvGGCYnTSY5NnnAj69qrqKssYx2V3sAatc7m8NGVEQUay9fS1xUXFDLHq6++q+vYnPYQl2NAdlY\nvhFwH+9gBosdh3fQ7Gxmk2MTxpigXtXYHDZmp87mhZUvBK1Mr5vevQm7wx70cgciHOdZHAQmd7k9\nybNt2Dr//PNZvHgx8+bN48EHH+Svf/0rt99+OwAJ0Qk8+uijfOc73wHgf//3f5k9ezYnnHACl19+\nOb/5zW96fM6E6AQAmpxNwXkRHnaHndy0XA0UA5BvyWdr1VbaOtpCXRWfGGM6v7iCHeS85R5uPUxx\nXXHQynW6nGyu2BzUwNiV1WJlf/1+KpsrQ1K+L8LxyuJzYKaI5OAOEpcBXx3qk/7s1a0UHqob6tMc\nIXdCMnedO6/f/R555BHS0tJobm5m6dKlvPPOO9yz7B5u+u+biIqI4rnnnuNHP/oRn3/+OS+++CKb\nNm2ivb2dRYsWsXjx4h6fMy4yjgiJoKk9eMGiraONgsoCLp9zedDKHAmsFitPFD5BYVVhyL6MBuJg\nw0EczQ7iIuOwO+xBPcPfWL6RuMg4WjpasDls5KQEZ0j4zsM7aXI2sciyKCjlded9X9gddk6belpI\n6tCfQA6dfQZYC8wWkRIRuVZELhCREmAZ8C8RecOz7wQR+TeAMcYJfAd4A9gG/MMYszVQ9QyGP/zh\nD+Tl5XHcccdx4MAB9uzZw8SpE9lm20ZVVRXbt29n+fLlrFmzhvPOO4+4uDiSkpI499xze31OESEh\nOiGoVxaFVYW0udqwWqxBK3Mk8P6/wr2Zwct7NXHhzAuDeobvvaI5efLJjI0dG9SrGm9ZoXpv56bn\nEhsZG9bNlYEcDdXb6efLPex7CDi7y+1/A//2Z318uQIIhPfff5+3336btWvXkpCQwIoVK6hvqufM\n88/k9Vdep6yojAsuuGBQZ27xUfE0NDVAkHprvG/kPEtecAocITLiM5icNBmbw8YqVoW6Ov2yO+yM\niR7DxbMu5untT2N32INyhn+o8RCOZgfWLCvNzuagBle7w05WQhbjx4wPWpldxUTGMC99XlifUIRj\nn8WIUltbS2pqKgkJCWzfvp1PP/2U1o5WTv1/p/L6v17nmWee4bLLLgNg+fLlvPrqq7S0tNDQ0MBr\nr73W53MnRLn7Ldo7gtPJbXPYmJI0hYz4jKCUN5JYLVbsFfYhjX4LFluFjYWZC5k2dhopsSlBO9vt\nenafb8mnuK6Y6pbqoJUd6itmq8VKYXUhLc7wnGSrwSLAzjzzTJxOJ3PnzuWOO+7guOOOo6WjhbS0\nNHLn5rJv3z6OOeYYAJYuXcrKlStZuHAhZ511FgsWLCAlJaXX546PigegzRX4jlNvE8FwaHMPR/mW\nfKpbqtlfvz/UVelTXVsdRYeLyLfkEyER5GfmBy1Y2B12EqMTmTl2ZlCb7kobSilvKg/5e9tqseJ0\nOSmoLOh/5xAIxw7uESU2Npb//Oc/R2zbdXgXsZGxPV453Hbbbfz0pz+lqamJk046qdcObnBP9oqL\niuNwx2G/17u74rpiDrceDlkH4HBnzXR/+W0s3xjWE682OTZhMJ3H2Wqx8kHJB1S3VJMWlxbQsm0O\nG3mZeURGRDIvYx7REdHYHDZOmXJKQMvd6HAPEw71e9sbrGwOG0vG9ZoNKWT0yiLInC4nbR1tnUNf\nu7vuuuvIz89n0aJFfOUrX2HRor7fwAnRCbS52gI+38J7hhfqS/XhatrYaSTHJGOvCN82aXB/UUVK\nJAsyFgDB65yva6tj1+FdnV+YsZGxzEufF5SrGpvDRkJUAjNTZwa8rL6kxKYwPWV62HZy65VFkHmH\nunr7G7p7+umnB/R8CVEJGGPYWb2TeRmB68S3OWykxKaQnZIdsDJGsgiJIN8SvCadwbJX2JmdNrvz\nZMZ7hm932AN6hr+5YjMGc8TJiNVi5cltT9La0UpsZGzAyrY77CzMXEhUROi/DvMt+by5701cxkWE\nhNe5fHjVZhRocjYhIn6b1OYNOoH+ErI5bORn5ofdG3g4sVqs7K3dS01LTair0qN2VztbKrYc8YUd\nGxlLbnpuUN5fkRLJwoyFndvyLfm0u9rZWhm4kfMNbQ3sqtkVNlfMVouV+rZ69tTsCXVVjqKf/CBr\ncjYRHxXvty/d6MhoIiMiA/phrm6ppriuOOQdgMNdfqZn4lWYNkXtqN5BS0fLUcfZarGytWorrR2t\nASvb7rAzK3XWEc2zXdvwA2VzxWZcxhU2721v0LJVhN8VqAaLIHIZFy3Ols5RTP4SExGDzWEL2LBM\n7a/wj/kZ84mKiArbpihvPihvZ7yX1WIN6Bl+u6udLZVbjnp/pcWlkZ2cHdD/10bHRiIkgrzM8Jg7\nNDlpMmlxadjKw+89osEiiJqdzRhjSIxO9OvzxkTGUNFcwcGGwKTQsjvsREdEMz9jfkCef7SIi4oj\nNy3wTTqDZa+wM3HMRLISs47YHugz/B3V7uSB1qyjT0a881NcxhWQsu0OO7NTZ/v9MzlYIoLVYg3L\n94gGiyDydm53v7JYvXo199xzz6CfNyYyBgjch9nmsHWmI1BDY7VY2VoZfkkFjTG9Zpj1nuEHakRU\n52S8zJ6DRW1rLcW1xX4v1+lysrkydMkDe2O1WClpKAm7pIIaLIKo2dlMTGTMUaMuVq5cyR13DHr1\nWKIkijHRYwLyYW7taGVr1VZtgvITq8VKm6uNwqrwWsvL++XU0xc2uK8uAjUD3eawMSFxwlFXNN5y\nvfv4mzcderi9tzv7LcLs6kKDRYAVFxczZ84cVq1axUmLTuL262/n7bffZvny5cycOZPPPvvsiBTl\nq1at4sYbb+T4449n2rRpvPBC/7n1RYS8zLyAdIoVVhXS7moPu7Ov4cqbVyvcvgi8Jxq9HWerxUpN\naw176/b6tdz+MgNkJ2eTGpsakP9XuPbFzU2bG5ZJBUM/sDhY/nMHlG3x73OOWwBn9d98VFRUxJPP\nPMmtv7qVK8+8kqeffpqPP/6Y1atX84tf/ILzzz//iP1LS0v5+OOP2b59OytXruSiiy7qt4x8Sz5/\nsf+FurY6kmMGvphSb7ydnt6RPGpoMuIzmJI0BZvDxtVcHerqdNro2EhSdBIzxs7o8f7Os91yG9NS\n/Lfc6cGGg1Q0V/T6hS0iAZufsrF8I+MSxzEucZzfn3sooiPd/YPhllRQryyCICcnh2lzpxEREcG8\n3HmceuqpiAgLFiyguLj4qP3PP/98IiIiyM3Npby83KcyrBYrBsMmxya/1t3usJOdnE16fLpfn3c0\ny7fkd64TES7sDjsLLQuJjIjs8f5AneH7kho8EAsDea9owu2qwstqsbKtahvNzuZQV6XT6Lmy8OEK\nIFBiY2Npbm8mMiKS6KhoYmPdHcURERE4nc4e9/fy9QtlQcYCIsU93+LESSf6pd7GGOwVdlZMXuGX\n51NuiyyLWL17NcV1xUFb3Kcvta21FNUUcWb2mb3uIyLkWfL8PkfE5rAxJnpMr1c08EUg2eTYxKlT\nT/VLuZ3p0MM4WDxsHqagsoCl45aGujqAXlkETZOzqdcUH/6QEJ3AnLQ5fv0w763bS01rTdh+oIar\ncFsMaVOF+2q0v+NstVjZV7ePquYqv5XdNXlgb3LTczvnEvmzXAi//gov77yPcOq30GARBAbTZ/JA\nf7FarGyp2OK3pIL9dXqqwclOyQ7qOhH9sTvsREpkv/NoOoOcn05I6trq2F2zu9/3V0xkDPMy5vl1\nAEfXdOjhKCU2hRljZ4TNewRGUzNUiGRnZ7N2w1oO1B8gISqBRx999Ij7CgrcuetXrVoFcMT9AA0N\nDT6XlW/J58ltT7KjeodfJtDZHDbGxo4lJzn0TSUjSbDXiejPRsdG5qTN6fdkJjc9tzOp4KlTht4c\n5E2H7svZfb4lnycKn6DF2eKXvGo2h42FGb330YSDfEs+b+x9I2ySCoa+BqOAv5MH9sb7ofOOYBoq\n7yStwSz5qvoW7JXgetPe0U5BZYFPX9ixkbHMz5jfuf7DUHVPh96XRZZFflsYyJsOvacZ4+HEarFS\n315PUU1RqKsCaLAIiqZ2/yYP7I0lwcLEMRP90kxQ1VzFvrp9YdumO9yFS7/FtupttHa0+nyc8y35\nFFb5Z+lPe4Xdpysa8G8Sxp7SoYcj7wTJUL9HvDRYBJjLuGjpaAlo53ZX3rwyQx2W6f1QhvsHaria\nnzG/s0knlAba0WvNdC/9ubVqaEkFe0qH3pexcWPJScnxS9NdT+nQw9GkpEmkx6WHTXOlBosA8yYP\nDHTntpfVYqWyuZKShpIhPY83eWBueq6faqa6CtY6Ef2xO9zJAzMTMn3a31/pN7ZXbe8xHXpfrBYr\ndsfQkwr2lA49HIkIi7IWhfw94qXBIsB6Sx4YKN4P31DPWG0OG/PS52nywAAKxjoRfTHGsNGxcUBX\nj6lxqX5JKjiYoav5mfnUtdWxt3bwKUd6S4cervIz892z3JsqQl0VDRaB1uRsIjYyNmhLNs4YO4Ok\n6KQhdUK2OFvcyQPDvANwuPOuBBeqpIIH6g9Q3VI94C9O79nuUM7wvenQLQmWAZULDOm93Vc69HAU\nTkkFAxYsROQREXGISEGXbWki8paI7PL8Tu3lsR0iYvf8rA5UHQPNGENTe1NQL3cjJIKFloVDOvPb\nWrUVp8vZawZS5R/eTlt/jV4bqMFOTBvqGX5f6dD7MiVpCmlxaUN6b/eVDj0czUmfQ1xk3MgOFsCj\nQPf8AXcA7xhjZgLveG73pNkYk+/5WRnAOgZUa0crLuMKWue2lzXTSlFNEbWttYN6vPeNqZPxAis9\nPp2pyVND1sltc9hIikli+tjpA3rcUM92venQF1kWDehxIjLk+Sl9pUMPR95Fx0Z0sDDGfAh0H0R+\nHvCY5+/HgPMZwZqcTRzcf5Bj849l1apVzJo1i6997WtHpSj/7LPPWLZsGVarleOPP54dO3YAcN99\n93HNNdcAsGXLFubPn09TU1O/5Xov171pHAbKmzwwNa7HCz/lR96V4EKRVNDusJOXmTfgId1Tk6e6\nl/4c5BfYUE5GrBYrB+oPDCqpYH/p0MOV1WJle/X2zv7PUAn2DO4sY0yp5+8yoLfwHici6wEncI8x\n5pWedhKR64DrAKZMmdJnwb/87Jdsr94+qEr3Zk7aHH5wzA96vd+bPHB30W5eeP4FHnnkEZYuXXpU\nivLHH3+cjz76iKioKN5++21++MMf8uKLL3LTTTexYsUKXn75Ze6++24eeOABEhL6v0qZnzGfKInC\n7rBz0qSTBvSaXMaFvcLOKZNPGdDj1OBYLVZeKXqFvXV7/Zr6uz+1rbXsrt3N2dPOHvBjveunDPaK\nyOaw9ZkOvS9dB3CcNvW0AT22v3To4cpqsdJhOiioLOCY8ceErB4h6+A27lOp3k6nphpjlgBfBX4n\nIj1eJxtjHjTGLDHGLMnM9G3oXzA1OhuJi4wjJyeHBQsWuFOUzzs6RXltbS0XX3wx8+fP55ZbbmHr\nVvcY9oiICB599FGuvPJKvvSlL7F8+XKfyo2PimdO2pxBdQTurd1LbWvtsPtADVf+Gr02UENd+GeR\nZdGg04Z706EPZpKqd3nfwby3wz15YG/yLHkIEvKmqGBfWZSLyHhjTKmIjAccPe1kjDno+b1HRN4H\nrMDuoRTc1xVAILS72mnvaCc+Ov6IlOMRERFHpSj/yU9+wsknn8zLL79McXExK1as6Nx/165djBkz\nhkOHDg2o/HxLPs/vfJ72jnaiI6N9ftxw/UANVznJOYyNHYvNYePCmRcGrVybw0aURA06h9hgz/B9\nSYfel5jIGOalzxtUcPUlHXo4So5JZvrY6SEPFsG+slgNXOX5+yrgn913EJFUEYn1/J0BLAfCa8Fi\nHzS3uxctiY/sf35FbW0tEydOBI5MJFhbW8uNN97Ihx9+SFVVlU9LrHpZLVZaO1rZVr1tQPW2OWyk\nxaUxNXnqgB6nBscfnbaDYXPYmJs+d9DzfwabNtzXdOh9GezCQL6kQw9XVouVTRWb6HB1hKwOgRw6\n+wywFpgtIiUici1wD3C6iOwCTvPcRkSWiMjDnofOBdaLyCbgPdx9FsMuWHiTB8ZG9T+p7fbbb+fO\nO+/EarUesRjSLbfcwg033MCsWbP429/+xh133IHD0ePF2FEGO2LF2+mpyQODx5rl/3Ui+tLW0cbW\nqq1D6uiNiYwZ1NKfQ72iAfd722kGllTQ13To4cpqsdLQ3hDSpIIBa4Yyxlzey11H5TY2xqwHvuH5\n+xOg/zSUYc6bPDAnJ6czDTnQa4rynTt3dm7/+c9/DsAjjzzSuW3y5MkUFfn+RslMyGTSmEnYHXau\nmndV/w8AKpsr2V+/n4tm9b/mt/KfrutE+CP1d38KqwoHlDywN/mWfB4vfJxmZ7PPVyg2h83n5IF9\nlQvuExtfV5EbSDr0cNQ18eTstNkhqYPO4A4Al3HR4gxe8sDeWC1WNjo2+jwsc6idnmpwuq4TEQz+\nOs4DTRvuTYc+1LP7lNgUpqdMH1An90DSoYejiWMmkhmf6dcFoAZKg0UANDubMQQveWBv8i35VLdU\nc6D+gE/72xw2YiJiNHlgkMVGxjIvfV7Q+i1sDhuTkyaTEZ8xpOcZ6EiugaZD76/sTRWbfE454o8r\nmlASEfIt+djKNVgETCgmOwUzeWBfr2+g/RZ2h535GfOJiYzxS92U77xJBf2xTkRfjDHYK+x++cJO\niU1hWso0n99f/hxpZ7VYqW+rZ3dN/4Mk212+L/AUzqwWK4caD1HeWB6S8kd0sIiLi6OqqiroAaPJ\n2URsVOCTBxpjqKqqIi6u5xX4po+dTlJMkk8f5hZnC4XVhcO2A3C4y7fk+2WdiP7sr99PdUu1346z\ndwa6L2f4A02H3l+54NuJ0GDSoYejztccoqaoEb0G96RJkygpKaGiInjpfQ2G8sZy4qLiaI9tD3h5\ncXFxTJo0qcf7vGs9+9JMUFBZ4E4eOMzPvoarrutELM5aHLByvEkL/ZVIL9+Sz4u7XmRPzR5mpPY+\nf8GbDn35BN8mlvZnctLkzqSCl8y+pM99R8rcodlps4mPisfusA96nspQjOhgER0dTU5OTlDL3HV4\nFzeuvpG7T7ibZdOXBbXsnlgtVj46+BG1rbWkxKb0up93ZTxvJlQVXGlxaX5ZJ6I/9go7STFJTBvr\nn9QiXc92+woW3nTo/jq7F5HOVSH7M5h06OEoOiKaBRkLQjY5b0Q3Q4VCuKVA9rUTcmP5RqalTGNs\n3NhgVEv1YCBNOoNlc9iwWqx+Ww/emza8v47XQJzdWy1WShpK+lwYyBjDxvKBLfAUzvIt+eyo3hGS\npIIaLPzM5rCRHpfOpKSem4aCzZtUsK+zEW/ywJHygRqurBYrta21FNcWB+T5a1pq2Fu716/H2dcz\n/MGmQ++LL/0WJfUlVLVUjZj3tjep4ObKzUEvW4OFn9kcNhZlLQqbGdDxUfHMTZ/b5wdqT80e6tvq\nh30H4HDn/f8PZSW4vgSqqdF7ht9XUsHBpkPvy9y0ucRGxvb53vZ2Bo+U93ZeZuiSCmqw8KOKpgoO\nNhwMu3b/fEs+W6u20t7Rc4e79wM1Us6+hqvs5GxSY1MD9kWw0bGRqIihpdroSdfO+Z5406EPdLGj\n/kRHRvebcmQo6dDDUVJMEjNSZ4RkwSwNFn4UrqMuFlkW0drRSmF1zym27A47aXFpTEnqe00QFVje\niVeB+iKwO+zkpuUSF9XzUOvByk3L7fMM3/t6AnF239/CQENJhx6urJmhSSo4cv6DYcDmsBEXGcec\n9DmhrsoROs/8eumE9HYAhkvT2WhmtVgHvU5EX9o62thauTUgJzLeM/ze3l/+SB7Ym76SCnrTofv7\niibUrFlWGtsbg55UUIOFH9kcNuZnzCc6wvf1I4IhIz6DyUmTezzzq2yupKShJOyuhkYr73HY5Bjc\nkri9KawqpM3VFrDj7D3D7ylt+FDTofclLzOvs4zu/JEOPRx5X0+g+rZ6o8HCT5ram9hevT1s35i9\nrfU8lPWQlf8Ndp2I/nifL8+S59fn9ertDL+9o33I6dD7khKbwoyxM3qc1RzIK5pQmpA4AUu8Jeid\n3Bos/KSgsoAO0xG2wcKbVHB//f4jttscNmIjY8lN0+SB4SAmMoZ5Gf5PKmhz2JiSNGXIyQN709sZ\nfmG1f9Kh9yXfks9mx+aj5qd4kwcGI0dbMAW6b6s3Giz8xOawIUjAztyGyttu2/3DbHfYmZc+b0BL\nr6rAslqsFFYX+i2poDEGu8Me0KtHb9rwnt5fENimIKvFSn17/RFt+P5Khx6urBYrpY2llDWWBa1M\nDRZ+YnPYmD52OskxyaGuSo9yUnJIjkk+4sPc7GxmW9U2FmWNrA7A4c5qsQ5onYj+FNcVc7j1cMA7\neq1ZVjY5jkwbvrF8o1/SofdZrndyXpcOdu8VzUh9b1uzvlgMKVg0WPhBh6uDTRWbwrYJCjxJBS1H\nrvVcUFmA02jywHDjnafjnUQ3VMFa1Kr7Gb4/06H3ZdKYSWTEZxzRbzHSF/KanepOKhjMfgsNFn5Q\nVFNEQ3tD2L8xrRYre2v3UtNSA3Tp9MwMz6az0Wps3FhyUnL89kVgc9hIiU0hOyXbL8/XG28+NO8X\ntb/ToffGm3Kk61m2zWHrDCIjUVREFAszFmqwGG6Gy1lM9zNWm8PG9JTpfWajVaHhzbfkj6SCNoeN\n/Mz8gE9Mm5Q0ifS49M4vsGAm1czPzOdgw0EcTQ6MMZ0JE0eyfEs+Ow7voLG9MSjlabDwA1uFjcz4\nTCaOmRjqqvRpfsZ8oiKiOr+ENlVsGrEdgMNdfmY+9W317KnZM6TnqW6ppriuOCjHuXtSQbvDv+nQ\n+9I1qaC/06GHK6vFisu42FwRnKSCGiz8wDvSJNxnQMdFxZGbnovdYWd3zW7q2+pHbAfgcOc9LkNd\nFc171RusWcxWi7XzDH+jY6Nf06H3ZU76HOIi47A77J3BaqTN3O5uYeZCBAlaJ7cGiyEqbyznYMPB\nYXPJa820UlBZwLrSdZ23VfjxrhMx1C8Cu8NOdEQ08zLm+almffN+Dt4/8L7f06H3JToimgWZC9jo\n2NiZDj0YVzShlBSTxKzUWUHrtwhYsBCRR0TEISIFXbalichbIrLL8zu1l8de5dlnl4hcFag6+sNw\ny9hqtVhpc7Xx7I5nw2rdDXUkESE/M3/IXwQ2h43cdHeiv2DwnuE/Xvg4ENyVF/Mz3QsDfVr6aVD6\naMJBviWfTRWbcLqcAS8rkP/NR4HuC8XeAbxjjJkJvOO5fQQRSQPuAo4FjgHu6i2ohAO7w058VDyz\n02aHuio+8U4a3Fe3T5MHhjmrxcqB+gODTirY2tHK1qrAJA/sTXSEO6ngvrp9AUmH3hfvwkDD6Up/\nqKwWK03OJnYd3hXwsgIWLIwxHwLV3TafBzzm+fsx4PweHvpl4C1jTLUx5jDwFkcHnbBhc9hYkLEg\n7JIH9iYjPqMzFflI7wAc7vpbJ6I/Wyu30u5qD/px9n5RByIdel/yLO6FgWD0vLd9WS3QX6ICXsKR\nsowxpZ6/y4CsHvaZCBzocrvEsy3sNLU3saN6B9cuuDbUVRmQfEs+++v3j5qzr+HKm1Twoc0P8WHJ\nhwN+vHd51mAvxuX9og72F3ZyTDLTx06nuLZ4xCUP7M34xPFYEizYHXa+OverAS0r2MGikzHGiIjp\nf8/eich1wHUAU6YEf+GevbV76TAd5KYPryR8504/l8rmSuamzw11VVQfYiJjuGDmBXxQ8gGfln46\nqOc4ferppMen+7lmfVuctZjFWYs5K+esoJYLcNGsi9hbu3fEJQ/sjYiwJGsJTc6eF3/ya1ndU1b7\n9clFsoHXjDHzPbd3ACuMMaUiMh543xgzu9tjLvfs8y3P7Qc8+z3TV1lLliwx69evD8Cr6N07+97h\n5vdv5rlznht2AUMpNTIYY4bU9ygiG4wxS/rbL9jDBVYD3tFNVwH/7GGfN4AzRCTV07F9hmdb2Clr\ncmd8HJc4LsQ1UUqNVsEapBJPrdNFAAAgAElEQVTIobPPAGuB2SJSIiLXAvcAp4vILuA0z21EZImI\nPAxgjKkG/hf43PPzP55tYaessYzYyFhSY8N2sJZSSvlFwPosjDGX93LXqT3sux74RpfbjwCPBKhq\nflPWWEZWQpYOP1VKjXgjf9ZKAJU1lmkTlFJqVNBgMQRlTRoslFKjgwaLQepwdVDRVEFWQk9TRZRS\namTRYDFIFc0VdJgOvbJQSo0KGiwGybtQugYLpdRooMFikHSOhVJqNNFgMUjljeWABgul1OigwWKQ\nyhrLSIhKICk6KdRVUUqpgNNgMUjeORY6IU8pNRr4FCxEZLqIxHr+XiEiN4rI2MBWLbzphDyl1Gji\n65XFi0CHiMwAHgQmA08HrFbDgE7IU0qNJr4GC5cxxglcAPzRGPN9YHzgqhXe2jvaqWquYlyCBgul\n1Ojga7Bo96wzcRXwmmfb8FhHNADKm8oxGL2yUEqNGr4Gi6uBZcDdxpi9IpIDPBG4aoU374S8rERN\n9aGUGh18SlFujCkEbuxyey/wy0BVKtzphDyl1GjTZ7AQkS1Ar+uuGmMW+r1Gw0Bnqg/ts1BKjRL9\nXVmc4/l9g+e3t+npCvoIIiNdWWMZyTHJJEQnhLoqSikVFH0GC2PMPgAROd0YY+1y1w9EZCNwRyAr\nF67KG8u1CUopNar42sEtIrK8y43jB/DYEUfnWCilRhtf1+C+Bvi7iKR4btd4to1KZY1lLMwYld01\nSqlRqt9gISIRwAxjTJ43WBhjagNeszDV7GymprVGryyUUqNKv01JxhgXcLvn79rRHChAU5MrpUYn\nX/sd3haR20RksoikeX8CWrMwpXMslFKjka99Fpd6ft/QZZsBpvm3OuFP51goFVhPr9vP5LR4TpyZ\nGeqqqC58urIwxuT08DPoQCEiN4lIgYhsFZGbe7h/hYjUiojd8/Pfgy3L3zTVh1KB8+HOCn748hau\nefRz3t1eHurqqC58vbJAROYDuUCcd5sx5vGBFuh5nm8CxwBtwOsi8poxpqjbrh8ZY8456glCrKyx\njLS4NGIiY0JdFaVGlKY2Jz98eQvTMhJJjI3i+ic38shVSzlhZkaoq6bwffGju4A/en5OBn4FrBxk\nmXOBdcaYJk/a8w+ACwf5XEGncywGb8O+ar706/e448XNtLR3hLo6Kszc++ZOSg43838XLuDxa45h\nWkYi33x8PZ/trQ511RS+d3BfBJwKlBljrgbygJS+H9KrAuBEEUkXkQTgbNyLKXW3TEQ2ich/RGTe\nIMvyu/LGcu2vGCBjDH9fs5dLH/iUprYOnv38AF+5/xP2VzWFumoqTGw6UMMja/bytWOncOy0dFIT\nY3ji2mMZPzaOax79HPuBmlBXMWBa2jv41evbeWNrWair0idfg0WzZwitU0SSAQc9f8H3yxizDXfG\n2jeB1wE70P00cyMw1RiTh/tq5pWenktErhOR9SKyvqKiYjDVGTBdTnVgGlud3PSsnZ+9WsiK2Rbe\n/t6XeGTVEg5UN3Hunz7mve2OUFdRhVh7h4sfvLiZzKRYfnDWnM7tmUmxPP2N40hLjOHrf1tHwcHA\njtqva2nnk92VGBO8tHfVjW1c8fA6/vL+br71xAbu/lchzg5X0MofCF+DxXrPmtsPARtwf5mvHWyh\nxpi/GWMWG2NOAg4DO7vdX2eMafD8/W8gWkSOarg0xjxojFlijFmSmRn4kRMNbQ00tDdosPDRnooG\nLvjLGl7bfIjvf3k2D165mJT4aE6Zk8Vr3z2RiWPjueaxz7n3rZ10uMI7L2W41284e+CD3Wwvq+fn\n5y8gOe7INdXGpcTx9DePZUxsFF9/5DN2ltcHpA6fFFVy5n0f8tWH1nHzc3aa2pwBKaervZWNXPiX\nNWw+WMt9l+bx9WVTeeijvXz14XU46lsCXv5AyUCjqIhkA8nGmM2DLlTEYoxxiMgU3FcYxxljarrc\nPw4oN8YYETkGeAH3lUavlV2yZIlZv379YKvkk6LDRVyw+gJ+ddKvOCvnrICWNdy9XlDKbc9vJiYq\ngj9cZu2xk7KlvYMfv1LACxtK+NKsTH53aT6pieExcKC+pZ3Pi6tZU1TFJ7ur2FZaR2JMJOljYkkf\nE0N6YiwZY2LI8N4eE0tGYkzn/akJMURGSKhfRtgrcjRw9u8/4vTcLP78tUW97ldc2cglD6zFAP/4\n1jJyMhL9Ur67CWgHj6zZy7SMRE6ZY+Fva/Yyy5LE/VcsYlrmGL+U09364mq++bj7++qhry9hSbZ7\n2tortoPc8dJmkuOi+cvXFnVuDyQR2WCMWdLvfr4ECxF5AvgQ9wil7X6o3EdAOtAOfM8Y846IXA9g\njPmriHwH+DbgBJo9+3zS13MGI1h8fPBjvv32t3n8rMexWqz9P2AUcna4+PWbO3jggz3kTR7L/V9b\nxISx8b3ub4zhmc8O8NPVW8lMiuWvVyxmwaTBdocNXkt7Bxv3HeaT3VWs2V3J5pJaOlyGmKgIlkxN\nxTplLC3tLqoaWqlsaKOyoZWqxjaqG9t6vOoQgbQEdzDJSIrh/y2YwMVLJhEdOWrzbx7F5TJc9uCn\n7Civ563vnYQlKa7P/XeV13Ppg58SGxXBP761jMlpQ1sioOBgLbc8Z2eXo4Grlk3ljrPmEh8TyYc7\nK7jpWRvODsOvL87jzPn+bUl4ddMhbn1+ExPHxvP3VUvJ7hb4tpXW8e0nN1ByuJkfnj2Xq5dnIxK4\nEw9/B4uTgRM9P9MBG/ChMeb3Q62ovwQjWLyw8wV+tvZnvPmVNxk/ZnxAyxqOKupbufEZG2v3VHHF\ncVP4yTm5xEZF+vTYTQdq+PaTG6hsbON/Vs7jsmOmBLSuzg4Xmw/WsnZ3FWuKKlm/7zBtTheREcLC\nSSksn57B8dPTWTQ1lbjo3l+Dy2WobW6n0hNEqhpbqWpoo6qhlQrP7+KqRnaWNzA1PYGbT5vJyryJ\netUBPPnpPn78SgG/umghlyzxrQu08FAdlz/0KcnxUfzjW8sYn9L7iUhvnB0u/vrBbn739i7Sx8Tw\n64vyOGnWkc3YJYeb+K+nNrK5pJbrvzSd286YRdQQA70xhvs/2M2vXt/B0uxUHrxySa9X0rXN7dz2\n/CbeKizn3LwJ3HPhAhJjfZ7pMCB+DRaeJ4wEluIeOns97k7vOX0/KniCESz+ZPsTD215iA1XbCAq\nIjAHbrjasO8w//XUBmqa2vnFBQv4yuJJA36O6sY2bnrWxke7Krl0yWR+dt68Pr+oB8LlMuwor2dN\nUSVrd1exbm81Da3udum545M5fno6x09P55icNJK6tZsPlTGGd7c7+M2bO9lWWsdMyxhuPWMWX543\nLqBnjOGsrLaF0+79gLzJKTx57bED+j9sOlDD1x5ehyUplue+tYzMpFifH1tc2cgt/7Bj21/DuXkT\n+N/z5jE2oecv7FZnBz97tZCn1+1n2bR0/vhVKxljfC+rq/YOF//9zwKe+ewA5+ZN4NcXLez3ve1y\nuYPLb9/cwfTMMfz1ysVMD0CzmL+vLN4BEnF3an8EfGyMCathLMEIFj/++MesLV3LOxe/E9ByhhNj\nDI+v3cfP/1XI+JR4/nrFYnInJA/6+Tpchvve2smf3iti/sRk7v/a4kE1Nxhj2FfVxJrdlXyyu4pP\nd1dR1dgGQE5GIsump7N8egbHTUsjfZBfAAPlchn+XVDKvW/tZE9FI/MnJnPrGbNZMStzVAUNYwzf\nfHwDHxdV8MbNJzE1feD9D58XV/P1v33GlLQEnrnOPWKqvzKfWrefu/+1jehI4ecXLGBl3gSfynp+\n/QF+/EoBqQkx/OWKRSyakjqguta3tHPD0zY+3FnBDSdP59bTZxMxgCvLNUWVfPcZG21OF7++aCFn\nLfBvq4a/g8V9wGKgFViDu/9irTGmeagV9ZdgBItvvPkNmp3NPHX2UwEtZ7hoanNy50tb+Kf9EKfO\nsXDvJfmkJPjnrPztwnJu+YedyAjhd5fms2K2pd/HlNW28IknOHxSVMmhWveIkqzkWJZPz2DZ9HSO\nn5HBxD76UILB2eHiZdtBfv/OLkoON7M0O5Vbz5jNcdPSQ1qvYPnX5lJueHojPzx7DtedNH3Qz/NJ\nUSWrHv2cWVljeOobx5ES3/N7z1HXwu0vbub9HRWcODODX1+Ux7iUvvtHutt6qJbrn9xAWW0LPzkn\nlyuPm+pTgD9U08w1j35OkaOBuy+Yz6VLB9e8eqimmf96aiP2AzVcd9I0bv/y7CE3i3n5vRnK86RJ\nwCrgNmCcMSY4p2Q+CEawOPflc5mVOovfrvhtQMvxxfriamqb2/nSrEy/vWkGYk9FA99+ciM7HfXc\nevos/mvFjAGdLfmiuLKR65/cwI7yem46dSY3njLziDJqmtpYu7uqs1N6T0UjAKkJ0Sybns6y6Rks\nn55OTkZiWJ65tzldPLf+AH96dxflda2cODOD286YTd7ksaGuWsDUNLVx2r0fMD4lnpf/6/ghv3ff\n2+HgusfXM39iCk9c6x5i29W/Npfyo1e20NLewZ1nzeXK46YO+n1a29TOzc/ZeG9HBRdYJ/KLCxYQ\nH9N7U1LBwVqufexzmlo7uP+KxUNOW9Lq7ODnr23jiU/3cdy0NP54+aIBNcH1xt9XFt/B3bm9GCjG\n3RT1kTHm3SHW028CHSyMMRzz1DFcMvsSvr/0+wErpz+2/Yf57Zs7+bioEoCJY+O5ctlULls6ude2\nV397vaCM7z+/iahI4feXWY/qHPSn5rYOfvTyFl6yHWTF7EyuOHYqnxVXs6aoksLSOoyBxJhIjslJ\n4/jpGRw/I52545L9HrgCqaW9gyfW7uP+D3ZT3djG6blZ3HrGLOaMG3xzXrj6/vObeMl2kNXfWc68\nCf4Z9fZ6QRk3PL2RxVNTeezqY4iPiaS2uZ27/lnAK/ZD5E1K4d5L8/3S3u9yGf70XhH3vb2T2VlJ\n/PWKxUeNZgJ4b7uDG57eyNj4aP5+9THMHpc05LK9XtpYwg9f3kJKvHt47eKpQxte6+9gcRvuALHB\nk88p7AQ6WNS01HDicydy+9LbuTL3yoCV05ttpXX89s0dvL3NQXpiDN9eMZ3JaQk8uqaYtXuqiIuO\n4ALrJK5ens2sLP+9Mb32VzXx6uZDvLrpENvL6lk4KYW/fG0Rk1KHNnzRF8YYnly3n/95dSvtHYaY\nyAgWTR3rHrE0I52Fk8aOiCGpDa1OHvl4Lw99uIeGNifnLpzALafP8tucglD7eFclV/xtHd9eMZ0f\nnOnfsTGrNx3ipmdtnDAjg2tOyOGHL23BUd/Kd0+ZwQ0nz/D7++MDz/DaDpfh3kvyOT33iyzUT6wt\n5q7VW8mdkMwjVy3FkjywJi9fbCut4/onN3DwcDM/+n9zWXX84IfXBmI01AnATGPM30UkExhjjNk7\nqNoFQKCDxfbq7Vz86sXcu+JeTp96esDK6W53RQP3vbWT1zaXkhQXxbdOmsbVy3OOGEa3rbSOxz4p\n5mXbQVqdLk6YkcGq47M5ZY5lSGfYjroWXttcyupNhzpz8yyaMpbz8idy6dLJfhup5KvdFQ2U17b0\nO5x1uKtpauPBD/fw9zXFtHW4OHPeOFbmT2DF7EyfhyKHm+a2Dr78uw+JjBD+c9OJATl+z68/wPdf\ncM8VnpaZyH2X5Ae0Se9AtXt47ZaDtdxw8nRuPm0Wv3p9Ow99tJfT5lr4/WXWgA13Bffw2lv/Yeft\nbQ5W5k3gd5fmD+rz7u8ri7uAJcBsY8wsEZkAPG+MWT7gmgVIoIPF+wfe57vvfpenz36aBZkLAlaO\n14HqJv7wzi5e3FhCXHQk1yzP4ZsnTuuzA7m6sY1nPtvPE2v3UVbXwtT0BK5als3FSyb5PBz0cGMb\nr28tY7X9EJ/urcIYyB2fzLl5Ezhn4fghT4RSvquob+WBD3bzsu0gVY1tJMVF8eV541iZN4Hjp6eH\npK9qsH7x7208+OEenr3uuIB25L9iO0iRo4EbTp7RZ3+Cv7S0d/CzV7fyzGcHsCTF4qhvZdXx2fzk\nnNygzKXxDq+ta2nnzrPmDuo5/B0s7IAV2GiMsXq2bTbGLBxU7QIg0MHi2e3Pcve6u3n34nfJTAhc\nG72jroU/vVfEM5/tR0S48ripfHvF9AGN727vcPHG1jL+vqaYDfsOkxgTycVLJnPV8dk9Nmk0tDp5\nq7CMVzeV8uHOCpwuw7SMRM7Nm8C5eeOZYfF/s5bynbPDxZrdVby66RBvFJRR3+okPTGGsxeMZ2X+\nBBZPSQ3rPpotJbWc9+ePuXTpFP7vwsCfaIXCPz4/wD2vb+eGk2dw7Qk5oa7OgPg7WHxmjDlGRDYa\nYxaJSCLuobOjJlj8bsPveKzwMTZcsYEI8f8ZXXVjG3/9YDePfVJMh8twydLJfPeUGYOaodrV5pIa\n/r6mmNc2H8LpMpw828LVy7NZmp3G+zscvLqplHe2l9PS7mJCSpwnQExg3oTksBxBNNq1tHfwwc4K\nVm86xDvbvjhu5+RN4NyFE5g/MbyOW3uHi5V/WkNVQytvfe9LvQ5vHQmMMWH1v/eVr8HC1wa1f4jI\nA8BYEfkmcA3w8FAqONyUNZWRlZDl90BR19LOwx/t5ZGP99LY5uSC/IncdNrMQU1U6snCSWO579J8\n7jxrDk+t289T6/Zx5d8+IypCcLoMGWNiuGTJZFbmTWBRmJ+hKoiLjuTL88bx5XnjaGx18va2clbb\nD/H3NXt58MM95HiuCFeGyRXhQx/tYVtpHQ94Mg6PZMMxUAzEQDq4TwfOAAR4wxjzViArNlCBvrJY\n9foqjDE8dtZjQ3qeDpdhX1UjO8vr2VxSy1Pr9lPb3M5Z88fxvdNnMTMAI5m6anV28NqmUgoO1XLK\nHAvLpg2vtm/Vs5qmNl4vKGP1pkOs3ePua5o7PplLl0ziymXZIclFtaeigTN//xGnzrFw/xWLg16+\n8k1AJuV1efII4HJjTNhMZQ50sDjzxTPJy8zjlyf90qf9jTGU1bWwvayenWX17CivZ0dZPUWOBlqd\n7sVNRGDFrExuPWM28ycGP9OqGpkcdS38a0sp/7S7R7EtnprKvZfk+e1q1Rcul+Hyhz5lW2kdb3/v\nSwEZPqr8wy/NUJ5V8W4AJgKrgbc8t28DNgFhEywCyWVclDeV97ro0eHGNnaU17PTExB2eIJDfcsX\nU1LGJccxa1wSx09PZ/a4ZGZnJTHDMiYoIzbU6GJJjuPq5TmsOj6bV+wH+e9/buWs33/ET87J5bKl\nk4PSXPLs5wdYt7eaX35lgQaKEaK/PosncK9ktxb4BvBD3M1Q5xtj7AGuW9iobqnG6XIeFSxqmto4\n789r2NdlLenkuCjmjEvm/PyJzBqXxOysJGZljQna7GqlvESEC6yTODYnndue38SdL23h7cJy/u8r\nC/pdO2Kw2pwunvt8P798fQfLpqX7nHpchb/+gsU0Y8wCABF5GCgFphhjwm/NvwAqa3QvpD4u4chg\nsXH/YfZVNXHdSdNYPiODOeOSsCTFjviOLjW8TBgbz5PXHsujnxTzy9e38+X7PuT/LlzAmfP9l73U\n2eHiJdtBfv/2Lg7WuJMj/vrihfpZGEH6Cxbt3j+MMR0iUjLaAgV0CRbdriwKD9UB8N1TZvh9DQSl\n/CkiQrjmhBxOnJnBLf+wc/2TG7lw0UR+unLeUeteD4TLZfjXllLue9uddn3hpBR+ceECTpqZoYFi\nhOkvWOSJSJ3nbwHiPbcFMMaYkZfprAe9BYuth+qYmp6ggUINGzOzknjp28v547u7+PN7RazbU81v\nLs5j2fSBzao2xvDONge/fcu9oNPsrCQeuHIxZ+RmaZAYofoMFsYY7X3FHSxiI2MZG3tknpnC0jrm\nDWGhH6VCISYqglvPmM3Jcyx87zk7X334U65dnsNtX57tU86mNUWV/PqNHdgP1JCdnsDvL8vnnIUT\ndKnYEU7XBvVBWVMZ4xKPXAKzvqWdfVVNXDyI5UOVCgeLpqTy75tO5Bf/3sbDH+/lw10V3HtJfq/D\nuDfsq+Y3b+xk7Z4qJqTEcc+F7uVzR0LGX9U/DRY+KGssO6pze1tpPcCQlhBVKtQSYqL4+fkLOG1u\nFre/sJkL/rKGm0+bxfVfmt55pVBwsJbfvrmD93ZUkDEmlrvOzeXyY6aM6My/6mgaLHxQ1ljGseOP\nPWJb4aFaAL8t4KJUKK2YbeGNm0/ix68U8Os3dvDudgc3nTqTZz/fz7+3lJESH80PzpzDVcdPJSFG\nvzZGIz3q/XC6nFQ0Vxw9Eqq0jvTEGCx+WNZQqXCQmhjDn75q5XR7Fj/5ZwFff+QzEmMiufHUmXzj\nxJwhjZpSw58Gi35UNlfiMq4eR0LlamZWNcKICOdbJ3JMThpvbi1jZf5E0hJ1QqmCkPRMichNIlIg\nIltF5OYe7hcR+YOIFInIZhFZFIp6Qs8T8tqcLnaVN2h/hRqxJoyNZ9XyHA0UqlPQg4WIzAe+CRwD\n5AHniMiMbrudBcz0/FwH3B/USnbR0xyL3RUNtHW4yB2vwUIpNTqE4spiLrDOGNNkjHECHwAXdtvn\nPOBx4/Yp7nU0/JebYAB6ChZbPTO3dY6FUmq0CEWwKABOFJF0EUkAzga6ZxubCBzocrvEsy3oyprK\nSIxOJCnmi3UmCg/VERcdQU7GmFBUSSmlgi7oHdzGmG0i8kvgTaARsAMdg3kuEbkOdzMVU6ZM8Vsd\nu+ppjkVhaS1zxiXrjFWl1KgRkg5uY8zfjDGLjTEn4U6BvrPbLgc58mpjkmdb9+d50BizxBizJDMz\nMyB1LWssO6IJyhhDoWcklFJKjRahGg1l8fyegru/4uluu6wGvu4ZFXUcUGuMKQ1yNYGjg0XJ4Wbq\nWpzaX6GUGlVCNc/iRRFJx50C/QZjTI2IXA9gjPkr8G/cfRlFQBNwdSgq2dbRRlVLFVmJWZ3bCkvd\nnds6EkopNZqEJFgYY07sYdtfu/xtcC/fGlLlTeXAkXMsth6qI0JgzjgNFkqp0UPTRfahp2GzhYfq\nmJapa2crpUYXDRZ96ClYbCut0yYopdSoo8GiD53NUJ5gcbixjYM1zToSSik16miw6ENZYxkpsSnE\nR8UD7qsK0JnbSqnRR4NFH7pPyPOOhJqrzVBKqVFGg0Ufus+x2HqojqzkWDLG6BoWSqnRRYNFH7xr\nb3sVHqrTlfGUUqOSBoteNDubqW2t7QwWLe0dFFU06EgopdSopMGiF95hs1kJ7tnbO8vr6XAZHQml\nlBqVNFj0ovsci0Jdw0IpNYppsOjFUcGitI4xsVFMTk0IZbWUUiokNFj0oqzpyGaorYfqmDs+iQhd\nw0IpNQppsOhFeWM56XHpxETG4HIZtpXqSCil1OilwaIXXedY7KtuoqmtQ0dCKaVGLQ0WvegaLLyd\n2zoSSik1Wmmw6EXXCXlbD9USFSHMzBoT4loppVRoaLDoQX1bPY3tjZ15oQpL65hhGUNslK5hoZQa\nnTRY9KCnORbaBKWUGs00WPSga7CoqG/FUd+qI6GUUqOaBoseeOdYjEsc15mWXEdCKaVGMw0WPShr\nLCNCIsiIz/hiJJQGC6XUKKbBogdljWVkxmcSFRHF1kO1TEqNJyUhOtTVUkqpkNFg0YPyxvIjckLp\nVYVSarTTYNED7xyLpjYneysbdSSUUmrUC0mwEJFbRGSriBSIyDMiEtft/lUiUiEids/PN4JVN2NM\n59rb20rrMQYdCaWUGvWCHixEZCJwI7DEGDMfiAQu62HX54wx+Z6fh4NVv5rWGlo7Wo8cCaVXFkqp\nUS5UzVBRQLyIRAEJwKEQ1eMoXedYFB6qIyU+mgkpcf08SimlRragBwtjzEHgN8B+oBSoNca82cOu\nXxGRzSLygohMDlb9jgwWtcybkIyIrmGhlBrdQtEMlQqcB+QAE4BEEbmi226vAtnGmIXAW8BjvTzX\ndSKyXkTWV1RU+KV+3gl5GXEWtpfV60gopZQiNM1QpwF7jTEVxph24CXg+K47GGOqjDGtnpsPA4t7\neiJjzIPGmCXGmCWZmZl+qVxZYxlREVHUNsTS6nRpf4VSShGaYLEfOE5EEsTdvnMqsK3rDiIyvsvN\nld3vD6SyxjKyErLYVtoA6EgopZQCd0dzUBlj1onIC8BGwAnYgAdF5H+A9caY1cCNIrLSc381sCpY\n9fMuelRYWkdMVATTMhODVbRSSoWtoAcLAGPMXcBd3Tb/d5f77wTuDGqlPMqbysm35FO4o47ZWUlE\nR+q8RaWU0m/CLlzGRXlTOeMSxrHVMxJKKaWUBosjVDVX4XQ5iY9I53BTu3ZuK6WUhwaLLrxzLFqa\nkwBNS66UUl4aLLrwzrGork1ABOZosFBKKSBEHdzhyntlUVIRS3Z6DGNi9d+jlFKgVxZHKGssIy4y\njp1lHdoEpZRSXWiw6KKssYzMhCwOVDdr57ZSSnWhwaKLsqYyxkSmA5qWXCmlutJg0UVZYxkRHakA\nzNNmKKWU6qQ9uB5Ol5PK5krGRCaRMSYWS7KuYaGUUl56ZeFR0VSBy7g4XJegTVBKKdWNBgsP7xyL\n8uo4HQmllFLdaLDw8M6xaG9N0ZxQSinVjQYLD2+wcDlTtBlKKaW60WDhUdZYRpTEEx+ZSHa6rmGh\nlFJdabDwKGssI7Ijlbnjk4iMkFBXRymlwooGC4+ypjJaW5K0CUoppXqgwcLjUH0pba0p5I7XNbeV\nUqo7DRZAW0cbNW2HMU4dCaWUUj3RYAGUN5a7/3CmMHtcUmgro5RSYUiDBV9MyBuXMJ646MgQ10Yp\npcKPBgu+mGMxJ2NSiGuilFLhSYMFsOfwQQDyJmSHtiJKKRWmNFgAO6sO4HImkDfREuqqKKVUWApJ\nsBCRW0Rkq4gUiMgzIhLX7f5YEXlORIpEZJ2IZAeyPvtrD2E0zYdSSvUq6MFCRCYCNwJLjDHzgUjg\nsm67XQscNsbMAO4DfhnIOlW2lBNDGmmJMYEsRimlhq1QNUNFAfEiEgUkAIe63X8e8Jjn7xeAU0Uk\nYDk4GjuqSI/VJiillEujEMcAAAf8SURBVOpN0FfKM8YcFJHfAPuBZuBNY8yb3XabCBzw7O8UkVog\nHaj0d30ONzdgIpo4u3Ed/P3/+fvplVIq8MYtgLPuCWgRoWiGSsV95ZADTAASReSKQT7XdSKyXkTW\nV1RUDKo+VY2NLGpLIy9Cl1FVSqnehGIN7tOAvcaYCgAReQk4Hniyyz4HgclAiaepKgWo6v5ExpgH\ngQcBlixZYgZTmRkZWTz2zQ8G81CllBo1QtFnsR84TkQSPP0QpwLbuu2zGrjK8/dFwLvGmEEFA6WU\nUkMX9GBhjFmHu9N6I7DFU4cHReR/RGSlZ7e/AekiUgR8D7gj2PVUSin1BRkpJ+xLliwx69evD3U1\nlFJqWBGRDcaYJf3tpzO4lVJK9UuDhVJKqX5psFBKKdUvDRZKKaX6pcFCKaVUv0bMaCgRqQD2DfLh\nGQQglUiY09c8OuhrHh2G8pqnGmMy+9tpxASLoRCR9b4MHRtJ9DWPDvqaR4dgvGZthlJKKdUvDRZK\nKaX6pcHC7cFQVyAE9DWPDvqaR4eAv2bts1BKKdUvvbJQSinVr1EfLETkTBHZISJFIjJistuKyGQR\neU9ECkVkq4jc5NmeJiJvicguz+9Uz3YRkT94/g+bRWRRaF/B4IhIpIjYROQ1z+0cEVnneV3PiUiM\nZ3us53aR5/7sUNZ7sERkrIi8ICLbRWSbiCwbBcf4Fs97ukBEnhGRuJF4nEXkERFxiEhBl20DPrYi\ncpVn/10iclVPZfliVAcLEYkE/gycBeQCl4tIbmhr5TdO4FZjTC5wHHCD57XdAbxjjJkJvMMX6d/P\nAmZ6fq4D7g9+lf3iJo5cH+WXwH3GmBnAYeBaz/ZrgcOe7fd59huOfg+8boyZA+Thfu0j9hiLyETg\nRmCJMWY+EAlcxsg8zo8CZ3bbNqBjKyJpwF3AscAxwF3eADNgxphR+wMsA97ocvtO4M5Q1ytAr/Wf\nwOnADmC8Z9t4YIfn7weAy7vs37nfcPkBJnk+QKcArwGCe6JSVPfjDbwBLPP8HeXZT0L9Ggb4elOA\nvd3rPcKP8UTgAJDmOW6vAV8eqccZyAYKBntsgcuBB7psP2K/gfyM6isLvnjjeZV4to0onktvK7AO\nyDLGlHruKgOyPH+PhP/F74DbAZfndjpQY4xxem53fU2dr9dzf61n/+EkB6gA/u5pentYRBIZwcfY\nGHMQ+A3uFTdLcR+3DYzs49zVQI+t3475aA8WI56IjAFeBG42xtR1vc+4TzVGxHA4ETkHcBhjNoS6\nLkEUBSwC7jfGWIFGuq0qOZKOMYCnCeU83IFyApDI0U01o0Kwj+1oDxYHgcldbk/ybBsRRCQad6B4\nyhjzkmdzuYiM99w/HnB4tg/3/8VyYKWIFAPP4m6K+j0wVkSiPPt0fU2dr9dzfwpQFcwK+0EJUGLc\nSxWDe7niRYzcYwxwGrDXGFNhjGkHXsJ97Efyce5qoMfWb8d8tAeLz4GZnpEUMbg7ylaHuE5+ISKC\ney3zbcaYe7vctRrwjoi4Cndfhnf71z2jKo4Dartc7oY9Y8ydxphJxphs3MfxXWPM14D3gIs8u3V/\nvd7/w0We/YfVGbgxpgw4ICKzPZtOBQoZocfYYz9wnIgkeN7j3tc8Yo9zNwM9tm8AZ4hIqueq7AzP\ntoELdQdOqH+As4GdwG7gR6Gujx9f1wm4L1E3A3bPz9m422vfAXYBbwNpnv0F98iw3cAW3KNNQv46\nBvnaVwCvef6eBnwGFAHPA7Ge7XGe20We+6eFut6DfK35wHrPcX4FSB3pxxj4GbAdKACeAGJH4nEG\nnsHdL9OO+yry2sEcW+Aaz+svAq4ebH10BrdSSql+jfZmKKWUUj7QYKGUUqpfGiyUUkr1S4OFUkqp\nfmmwUEop1S8NFkp1IyIdImLv8tNnNmIRuV5Evu6HcotFJGOoz6NUIOjQWaW6EZEGY8yYEJRbjHt8\nfGWwy1aqP3ploZSPPGf+vxKRLSLymYjM8Gz/qYjc5vn7RnGvIbJZRJ71bEsTkVc82z4VkYWe7eki\n8qZnbYaHcU+s8pZ1hacMu4g8IO51OiJF5FHPOg5bROSWEPwb1CilwUKpo8V3a4a6tMt9tcaYBcCf\ncGe57e4OwGqMWQhc79n2M8Dm2fZD4HHP9ruAj40x84CXgSkAIjIXuBRYbozJBzqAr+GerT3RGDPf\nU4e/+/E1K9WnqP53UWrUafZ8SffkmS6/7+vh/s3AUyLyCu70G+BOvfIVAGPMu54rimTgJOBCz/Z/\nichhz/6nAouBz93pj/5/e/evWkUQxXH8+4vVhYDEIl0I2NiKT5A2pA0E8go+gARucX0FbSUgCFZW\nEgi3SUggaSWgTyFoFQkpjsXsxYWgm7uIjd8PLPtn2IVp9swZhjNMaAXjPgKPk7wGjoD5+C5KyzGz\nkJZTv7le2KHV6HlG+9mPGZAFeFtVT7vjSVXNquobbTe8U1rW8mbEt6VRDBbScvZ658t+Q5IVYKOq\nToAXtHLYq8A5bRqJJFvA12p7i5wB+93zbVoRQGiF4naTrHdtj5JsdiulVqrqAzClBSTpn3AaSrpr\nkuRT7/64qhbLZ9eSXAE3tC0r+x4A75I8pGUHr6rqe5IZcNi9d82vEtMvgfdJPgMXtPLbVNWXJFNg\n3gWgW+A58IO2K95ikHfw97os/ZlLZ6V7cmmr/mdOQ0mSBplZSJIGmVlIkgYZLCRJgwwWkqRBBgtJ\n0iCDhSRpkMFCkjToJ1qXJQO1U4YbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['avg'], label = 'avg')\n",
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['min'], label = 'min')\n",
    "plt.plot(aggr_ep_rewards['ep'], aggr_ep_rewards['max'], label = 'max')\n",
    "plt.legend(loc='upper left')\n",
    "plt.xlabel('Episodes')\n",
    "plt.ylabel('Rewards')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nwD41YstHyGx"
   },
   "source": [
    "### Save model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YU3N0zE_HxQb"
   },
   "outputs": [],
   "source": [
    "agent.model_prediction.save('dqn_1.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9ILbxFFfIckN"
   },
   "source": [
    "### Rendering Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iFZ1eIgjHwXw"
   },
   "outputs": [],
   "source": [
    "done = False\n",
    "state = env.reset()\n",
    "while not done:\n",
    "    qs_list = agent.get_qs(state)\n",
    "    action = np.argmax(qs_list())\n",
    "    next_state, _, done, _ = env.step(action)\n",
    "    state = next_state\n",
    "    env.render()\n",
    "\n",
    "env.close()\n",
    "    "
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "1_dqn_kearas_cartpole.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python (spinningup)",
   "language": "python",
   "name": "spinningup"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
